Marking 3D locations from ultrasound images

ABSTRACT

A method for marking three-dimensional (3D) locations from images obtained from an ultrasound imaging system including a transducer, comprising the steps of: tracking the pose of the transducer with respect to an external 3D coordinate system; obtaining a two-dimensional (2D) ultrasound image from the transducer; marking a desired target with a marker on the 2D ultrasound image; and calculating the 3D position of the marker utilizing data from the step of tracking.

[0001] Reference is hereby made to the following U.S. Provisional patentapplications whereof the benefit is hereby claimed and the disclosureshereby incorporated by reference:

[0002] U.S. Provisional patent application No. 60/312.872, entitledMARKING 3D LOCATIONS FROM ULTRASOUND IMAGES and filed Aug. 16, 2001 inthe names of Frank Sauer, Ali Khamene, Benedicte Bascle;

[0003] U.S. Provisional patent application No. 60/312,876, entitledLOCAL 3D RECONSTRUCTION FROM ULTRASOUND IMAGES and filed Aug. 16, 2001in the names of Frank Sauer, Ali Khamene, Benedicte Bascle;

[0004] U.S. Provisional patent application No. 60/312,871, entitledSPATIOTEMPORAL FREEZING OF ULTRASOUND IMAGES IN AUGMENTED REALITYVISUALIZATION and filed Aug. 16, 2001 in the names of Frank Sauer, AliKhamene, Benedicte Bascle;

[0005] U.S. Provisional patent application No. 60/312,875, entitled USERINTERFACE FOR AUGMENTED AND VIRTUAL REALITY SYSTEMS and filed Aug. 16,2001 in the names of Frank Sauer, Lars Schimmang, Ali Khamene; and

[0006] U.S. Provisional patent application No. 60/312,873, entitledVIDEO-ASSISTANCE FOR ULTRASOUND GUIDED NEEDLE BIOPSY and filed Aug. 16,2001 in the names of Frank Sauer and Ali Khamene.

[0007] Reference is hereby made to the following copending U.S. patentapplications being filed on even date herewith.

[0008] U.S. patent application entitled LOCAL 3D RECONSTRUCTION FROMULTRASOUND IMAGES and filed in the names of Frank Sauer, Ali Khamene,Benedicte Bascle;

[0009] U.S. patent application entitled SPATIOTEMPORAL FREEZING OFULTRASOUND IMAGES IN AUGMENTED REALITY VISUALIZATION and filed in thenames of Frank Sauer, Ali Khamene, Benedicte Bascle;

[0010] U.S. patent application entitled USER INTERFACE FOR AUGMENTED ANDVIRTUAL REALITY SYSTEMS and filed in the names of Frank Sauer, LarsSchimmang, Ali Khamene; and

[0011] U.S. patent application entitled VIDEO-ASSISTANCE FOR ULTRASOUNDGUIDED NEEDLE BIOPSY and filed in the names of Frank Sauer and AliKhamene.

[0012] The present invention relates to imaging and, more specificallyto ultrasound imaging.

[0013] Ultrasound scanners are commonly utilized to capture live 2Dimages from within objects or patients. Scanners typically have astandard option to freeze an image in time and display the still imageon the screen for evaluation, e.g. for measuring spatial dimensions inthe image.

[0014] Helpful background material on augmented reality and relatedtopics can be found in Proceedings of the IEEE and ACM InternationalSymposium on Augmented Reality 2000, dated Oct. 5-6, 2000; Munich,Germany; IEEE Computer Society, Los Alamitos, Calif., U.S.A. In theabove-cited Proceedings, an article of particular interest entitledAUGMENTED WORKSPACE: DESIGNING AN AR TESTBED is published on pages47-53, and is authored by Frank Sauer, an inventor in the presentapplication, et alii.

[0015] See also the review article by R. T. Azuma: “A Survey ofAugmented Reality”, Presence: Teleoperators and Virtula Environments,6(4), 355-386, (1997).

[0016]FIG. 1 show a schematic block diagram of such an augmented realitysystem as may be utilized in conjunction with features of the invention.A tracker camera 10 is coupled by way of and A/D (analog to digital)converter 12 to a programmable digital computer 14. Two scene cameras 16are coupled to computer 14. An ultrasound scanner 16, having atransducer 18, is coupled by way of an A/D converter 20 to computer 14.A head-mounted display (HMD) control unit 22 is coupled for signalinterchange with computer 14 and to an HMD display 24.

[0017] In accordance with an aspect of the present invention, a methodis provided for marking (or outlining) of targets on an ultrasound slicein the 2-dimensional (2D) ultrasound slice plane and in 3 dimensions(3D).

[0018] In accordance with another aspect of the present invention, asystem tracks the pose of the transducer used in the ultrasound scanner,and hence the pose of the ultrasound slice, with respect to a fixedexternal 3D coordinate system.

[0019] In accordance with another aspect of the present invention, asystem provides an interface to the user that enables the user either todirectly mark a desired target in an ultrasound image, or trigger thesystem to (semi)automatically locate a target in a given ultrasoundimage.

[0020] In accordance with another aspect of the invention, a method formarking three-dimensional (3D) locations from images obtained from anultrasound imaging system including a transducer, comprising the stepsof: tracking the pose of the transducer with respect to an external 3Dcoordinate system; obtaining a two-dimensional (2D) ultrasound imagefrom the transducer; marking a desired target with a marker on the 2Dultrasound image; and calculating the 3D position of the markerutilizing data from the step of tracking.

[0021] The invention will be more fully understood from the followingdetailed description of preferred embodiments, in conjunction with theDrawing in which the FIG. 1 shows a system block diagram of an augmentedreality system.

[0022] In accordance with a preferred embodiment, an ultrasound scanner,including a transducer, is utilized for imaging. The user is providedwith augmented reality visualization, where the ultrasound slices areoverlaid onto a view of the patient/object in a registered manner.Tracking apparatus is provided for tracking the transducer and fortracking the user's viewpoint for the augmented reality application.Such augmented reality application techniques, whereby structures thatare visible in the ultrasound images appear in the location of thecorresponding physical structures are, per se, known in the art and neednot be set forth in detail.

[0023] Preferably, the augmented view is stereoscopic to provide theuser with depth perception. A calibration procedure is provided and theequipment is coupled to a computer for generating the graphics requiredand providing the necessary image processing.

[0024] Furthermore, a local user interface is provided in accordancewith the principles of the present invention.

[0025] With the user interface, a user can initiate the targetlocalization and marking.

[0026] In accordance with the present embodiment, the user interface isequipped with a pointing device, such as a computer-type mouse, coupledto the computer.

[0027] In operation, a user can use the pointing device, such as amouse, to mark a 2D location in the image. With the knowledge of the 3Dpose of that image, the system calculates the 3D position of the marker.The user's input may be in an “on-line” mode, where he holds thetransducer still in the desired pose, or in an “off-line” mode, wherethe user first freezes, that is, records the image, together with itspose information, and then places the marker in the recorded still imageat his leisure. For the on-line mode, the pointing device is preferablyattached to the transducer and can be operated with the same hand thatis holding the transducer, or alternatively, it is placed on the floorand operated by way of a foot-pedal.

[0028] In accordance with another preferred embodiment, the systemperforms image processing on an ultrasound image and automatically, orsemi-automatically, locates a target of a type optionally predeterminedby the user. The input required of the user is thereby simplified. Forexample, without an extra pointing device, the user may place thetransducer in a pose where the target structure appears on the verticalcenterline of the ultrasound image. The user then triggers the system tolocate the target along the image's centerline such as by using a buttonon the transducer, by operating a foot-switch, by using voice control,or any other convenient control device. The system in accordance withthe present embodiment includes a processor that searches the imagealong its centerline, which makes locating the target easier than if thesearch would have to be conducted over the whole image.

[0029] A preferred search algorithm would be to de-noise the imagearound the centerline, for example, with a median filter, identifypotential target locations in a line scan along the centerline, andverify the existence of a target with, for example, a Hough transform.The Hough transform is known and may be found in various textbooks, suchas, for example, “Fundamentals of Electronic Image Processing” by ArthurR. Weeks, Jr., IEEE Press, New York, N.Y.; 1996.

[0030] When the system proposes a target location, the user then acceptsor rejects the proposed location with another trigger input by using abutton on the transducer, a footswitch, voice control, etc.

[0031] As an alternative to using a line on the ultrasound image aspointer, a line other than the vertical centerline can be used, such asa vertical off-center line, or a line that is tilted at an angle withrespect to the vertical direction.

[0032] A target position along a line can be input by user via athumbwheel at transducer or a similar one-dimensional (1D) pointingdevice for which no image processing is necessary.

[0033] In an alternative embodiment, the user uses a line on theultrasound image to point to target from two different transducer poses,so that the system can calculate the target location as the intersectionof the two lines in 3D space. Alternatively two different lines can beused to require less movement between the two pointing transducer poses,for example, two lines that intersect in the image.

[0034] The image processing is powerful enough to find a target in the2D image. The user need then only position the transducer so that thetarget is visible in the image, initiate the automatic target search,and then confirm or reject a target proposed by the system. Severaltargets can be found in the same image. This embodiment is attractivefrom a user point of view as it requires the least input; however,robust target detection in ultrasound images is difficult. Therefore,there are advantages to providing the user with a 1D or 2D pointinginterface as described in the above embodiments.

[0035] The system thus graphically marks confirmed target locations andrecords the respective 3D coordinates thereof. Preferred visualizationis in a stereoscopic augmented or virtual reality fashion.

[0036] Instead of just marking the target location as the target center,the system can also perform image processing to obtain an outline of thetarget, and show this outline in a graphical representation.

[0037] Marking of the 3D locations of target structures can be helpfulfor diagnostic purposes, for guidance of interventional procedures, aswell as for registering the ultrasound images to images taken with otherimaging modalities like CT (computer tomography) and MR. (magneticresonance imaging).

[0038] While the invention has been explained by way of exemplaryembodiments, it will be understood by one of skill in the art to whichit pertains that various modifications and changes may be readily madewithout departing from the spirit of the invention which is defined bythe claims following. For example, where the claims refer to marking atarget, this is to be understood to encompass deriving an outline of atarget by image processing to present the outline by graphicalrepresentation.

What is claimed is:
 1. A method for marking three-dimensional (3D)locations from images obtained from an ultrasound imaging systemincluding a transducer, comprising the steps of: tracking the pose ofsaid transducer with respect to an external 3D coordinate system;obtaining a two-dimensional (2D) ultrasound image from said transducer;marking a desired target with a marker on said 2D ultrasound image; andcalculating the 3D position of said marker utilizing data from said stepof tracking.
 2. A method for marking as recited in claim 1, wherein saidstep of marking comprises marking said desired target while keeping saidtransducer stationary.
 3. A method for marking as recited in claim 2,wherein said step of marking comprises depressing a switch to initiatesaid marking.
 4. A method for marking as recited in claim 2, whereinsaid step of depressing a switch to initiate said marking comprisesdepressing a switch attached to said transducer.
 5. A method for markingas recited in claim 2, wherein said step of depressing a switch toinitiate said marking comprises depressing a foot-switch.
 6. A methodfor marking as recited in claim 2, wherein said step of markingcomprises actuating a voice recognition system to initiate said marking.7. A method for marking as recited in claim 1, wherein said step ofmarking comprises making a recorded image of said 2D ultrasound image,recording appurtenant pose data from said step of tracking; and markingsaid recorded image.
 8. A method for marking as recited in claim 7,wherein said step of marking comprises depressing a switch to initiatesaid marking.
 9. A method for marking as recited in claim 8, whereinsaid step of depressing a switch to initiate said marking comprisesdepressing a switch attached to said transducer.
 10. A method formarking as recited in claim 8, wherein said step of depressing a switchto initiate said marking comprises depressing a foot-switch.
 11. Amethod for marking as recited in claim 8, wherein said step of markingcomprises actuating a voice recognition system to initiate said marking.12. A method for marking three-dimensional (3D) locations from imagesobtained from an ultrasound imaging system including a transducer,comprising the steps of: tracking the pose of said transducer withrespect to an external 3D coordinate system; obtaining a two-dimensional(2D) ultrasound image from said transducer; adjusting said pose of saidtransducer such that said target coincides with a designated part ofsaid 2D ultrasound image for indicating marker location; automaticallylocating a potential target of a predetermined type in said designatedpart; marking said target with a marker; and calculating the 3D positionof said marker utilizing data from said tracking step.
 13. A method formarking as recited in claim 12, wherein said step of adjusting said poseof said transducer comprises adjusting said pose of said transducer suchthat said target is on a center-line of said 2D ultrasound image.
 14. Amethod for marking as recited in claim 1, wherein said step ofautomatically locating a potential target comprises using a searchalgorithm for denoising said 2D ultrasound image around saidcenter-line.
 15. A method for marking as recited in claim 13, whereinsaid step of automatically locating a potential target comprises using asearch algorithm for de-noising said 2D ultrasound image around saidcenter-line by using a median filter.
 16. A method for marking asrecited in claim 13, wherein said step of automatically locating apotential target comprises using a search algorithm for denoising said2D ultrasound image around said center-line by using a median filter.17. A method for marking as recited in claim 13, wherein said step ofautomatically locating a potential target comprises scanning along saidcenter-line and verifying the existence of a target by using a Houghtransform.
 18. A method for marking as recited in claim 12, wherein saidstep of adjusting said pose of said transducer comprises adjusting saidpose of said transducer such that said target coincides with a verticaloff-center line in said 2D ultrasound image for indicating markerlocation.
 19. A method for marking as recited in claim 12, wherein saidstep of adjusting said pose of said transducer comprises adjusting saidpose of said transducer such that said target coincides with a linetilted at an angle in said 2D ultrasound image for indicating markerlocation.
 20. A method for marking as recited in claim 12, wherein saidstep of adjusting said pose of said transducer comprises adjusting saidpose of said transducer such that said target is on a designated line insaid 2D ultrasound image for indicating marker location.
 21. A methodfor marking as recited in claim 20, including a step of inputting targetposition along said designated line by a pointing device.
 22. A methodfor marking as recited in claim 20, wherein said step of inputtingtarget position comprises inputting target position by operating a thumbwheel.
 23. A method for marking as recited in claim 12, wherein saidstep of adjusting said pose of said transducer comprises adjusting firstand second poses of said transducer such that said target is on adesignated line for indicating target location in said 2D ultrasoundimage for each pose.
 24. A method for marking as recited in claim 12,wherein said step of adjusting said pose of said transducer comprisesadjusting said pose of said transducer such that said target is on firstand second designated intersecting lines in said 2D ultrasound image forindicating target location.
 25. A method for marking three-dimensional(3D) locations from images obtained from an ultrasound imaging systemincluding a transducer, comprising the steps of: tracking the pose ofsaid transducer with respect to an external 3D coordinate system;obtaining a two-dimensional (2D) ultrasound image from said transducer;automatically locating a potential target of a predetermined type insaid 2D ultrasound image; and optionally marking said target with amarker; and calculating the 3D position of markers marked with a marker,utilizing data from said tracking step.
 26. A method for markingthree-dimensional (3D) locations from images obtained from an augmentedreality (AR) ultrasound imaging system including a transducer,comprising the steps of: tracking the pose of said transducer withrespect to an external 3D coordinate system from the viewpoint of auser; obtaining a two-dimensional (2D) ultrasound image of an objectfrom said transducer; overlaying said ultrasound image onto a view ofsaid object in registration therewith from said viewpoint; marking adesired target with a marker on said 2D ultrasound image; andcalculating the 3D position of said marker utilizing data from said stepof tracking.
 27. A method for marking three-dimensional (3D) locationsfrom images obtained from a stereoscopic augmented reality (AR)ultrasound imaging system including a transducer, comprising the stepsof: tracking the pose of said transducer with respect to an external 3Dcoordinate system from the viewpoint of a user; obtaining atwo-dimensional (2D) ultrasound image of an object from said transducer;overlaying said ultrasound image onto a view of said object inregistration therewith from said viewpoint of a user; marking a desiredtarget with a marker on said 2D ultrasound image; calculating the 3Dposition of said marker utilizing data from said step of tracking; andproviding stereoscopic visualization of said marker utilizing said 3Dposition of said marker.
 28. Apparatus for marking three-dimensional(3D) locations from images obtained from an ultrasound imaging systemincluding a transducer, comprising: means for tracking the pose of saidtransducer with respect to an external 3D coordinate system; means forobtaining a two-dimensional (2D) ultrasound image from said transducer;means for marking a desired target with a marker on said 2D ultrasoundimage; and means for calculating the 3D position of said markerutilizing data from said step of tracking.
 29. Apparatus for marking asrecited in claim 28, wherein said means for marking comprises a switchto initiate said marking.
 30. Apparatus for marking as recited in claim29, wherein said switch is attached to said transducer.
 31. Apparatusfor marking as recited in claim 29, wherein said switch comprises afoot-switch.
 32. Apparatus for marking as recited in claim 28, whereinsaid means for marking comprises a voice recognition system to initiatesaid marking.
 33. Apparatus for marking as recited in claim 28, whereinsaid means for marking comprises means for making a recorded image ofsaid 2D ultrasound image and recording appurtenant pose data from saidmeans for tracking and said means for marking includes means for markingsaid recorded image.
 34. Apparatus for marking three-dimensional (3D)locations from images obtained from an ultrasound imaging systemincluding a transducer, comprising: means for tracking the pose of saidtransducer with respect to an external 3D coordinate system; means forobtaining a two-dimensional (2D) ultrasound image from said transducer;means for adjusting said pose of said transducer such that said targetcoincides with a designated part of said 2D ultrasound image forindicating marker location; means for automatically locating a potentialtarget of a predetermined type in said designated part; means formarking said target with a marker; and means for calculating the 3Dposition of said marker utilizing data from said tracking step. 35.Apparatus for marking as recited in claim 34, wherein said designatedpart comprises a center-line of said 2D ultrasound image.
 36. Apparatusfor marking as recited in claim 34, wherein said means for automaticallylocating a potential target comprises a search algorithm for de-noisingsaid 2D ultrasound image around said center-line.
 37. Apparatus formarking as recited in claim 34, wherein said means for automaticallylocating a potential target comprises a search algorithm for de-noisingsaid 2D ultrasound image around said center-line by using a medianfilter.
 38. Apparatus for marking as recited in claim 34, wherein saidstep of automatically locating a potential target comprises means forscanning along said center-line and verifying the existence of a targetby using a Hough transform.
 39. Apparatus for marking as recited inclaim 34, wherein said wherein said designated part comprises a verticaloff-center line in said 2D ultrasound image for indicating markerlocation.
 40. Apparatus for marking as recited in claim 34, wherein saiddesignated part comprises a line tilted at an angle in said 2Dultrasound image for indicating marker location.
 41. Apparatus formarking as recited in claim 34, wherein said designated part comprises adesignated line in said 2D ultrasound image for indicating markerlocation.
 42. Apparatus for marking as recited in claim 41, including apointing device for inputting marker location.
 43. A method for markingas recited in claim 42, wherein said pointing device comprises athumbwheel.
 44. Apparatus for marking three-dimensional (3D) locationsfrom images obtained from an ultrasound imaging system including atransducer, comprising: means for tracking the pose of said transducerwith respect to an external 3D coordinate system; means for obtaining atwo-dimensional (2D) ultrasound image from said transducer; means forautomatically locating a potential target of a predetermined type insaid 2D ultrasound image; means for optionally marking said target witha marker; and means for calculating the 3D position of markers markedwith a marker, utilizing data from said tracking step.
 45. Apparatus formarking three-dimensional (3D) locations from images obtained from anaugmented reality (AR) ultrasound imaging system including a transducer,comprising: means for tracking the pose of said transducer with respectto an external 3D coordinate system from the viewpoint of a user; meansfor obtaining a two-dimensional (2D) ultrasound image of an object fromsaid transducer; means for overlaying said ultrasound image onto a viewof said object in registration therewith from said viewpoint; means formarking a desired target with a marker on said 2D ultrasound image; andmeans for calculating the 3D position of said marker utilizing data fromsaid step of tracking.
 46. Apparatus for marking three-dimensional (3D)locations from images obtained from a stereoscopic augmented reality(AR) ultrasound imaging system including a transducer, comprising: meansfor tracking the pose of said transducer with respect to an external 3Dcoordinate system from the viewpoint of a user; means for obtaining atwo-dimensional (2D) ultrasound image of an object from said transducer;means for overlaying said ultrasound image onto a view of said object inregistration therewith from said viewpoint of a user; means for markinga desired target with a marker on said 2D ultrasound image; means forcalculating the 3D position of said marker utilizing data from said stepof tracking; and means for providing stereoscopic visualization of saidmarker utilizing said 3D position of said marker.